Head mounted display and setting method

ABSTRACT

In a head mounted display  100 , a memory  71  stores an application. An image pickup unit  74  takes an image of a site of a user  1 , and a position specifying unit  73  specifies a position and a direction of the head mounted display  100 . A detector  75  detects a position indicated by the user  1  on the basis of the image taken by the image pickup unit  74 , and a setting unit  76  sets a position indicating a home position on the basis of a result detected by the detector  75  and the position and the direction specified by the position specifying unit  73.

TECHNICAL FIELD

The present invention relates to a head mounted display and a settingmethod.

BACKGROUND ART

There is a technique for displaying a plurality of windows on a displayof an information processing apparatus and switching the windows inresponse to an operational input (for example, Patent document 1).

RELATED ART DOCUMENTS Patent Documents

-   Patent document 1: Japanese Patent Application Publication No.    2015-43135

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In the technique disclosed in Patent document 1 described above, thewindows whose positions are fixed and that are displayed on the displayof the information processing apparatus are controlled.

Here, a head mounted display in which an absolute coordinate fordisplaying an application (or a window in which the application isdisplayed) is defined is supposed. In this case, when a user operatesthe application, it may be necessary for the user to move in accordancewith a position associated with an application to be used. As a result,the operation may become complicated.

It is an object of the present invention to provide a head mounteddisplay by which an application can be operated easily.

The foregoing and other objects, and new features of the presentinvention will become more apparent from the detailed description of thepresent specification and the appending drawings.

Means for Solving the Problem

Techniques described in the claims are used as means for solving theproblem.

In one example, there is provided a head mounted display storing anapplication. The head mounted display includes: an image pickup unitconfigured to take an image of a site of a user who wears the headmounted display; a detector configured to detect a position indicated bythe user on a basis of the image taken by the image pickup unit; aposition specifying unit configured to specify a position and adirection of the head mounted display; and a setting unit configured toset information regarding an operation area on a basis of a resultdetected by the detector and the position and the direction specified bythe position specifying unit, the operation area being an area where theuser operates the application.

Effects of the Invention

By using the technique of the present invention, it is possible tooperate an application easily.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIGS. 1(a)-1(c) are views illustrating an overview of a head mounteddisplay according to the present embodiment;

FIG. 2 is a view for explaining a visible area;

FIG. 3 is a view for explaining a hardware configuration of the headmounted display;

FIG. 4 is a functional block diagram of the head mounted display;

FIGS. 5(a)-5(b) are views for explaining a method of setting a homeposition;

FIGS. 6(a)-6(b) are views for explaining another method of specifying ahome position;

FIGS. 7(a)-7(b) are views for explaining a method of calculating adistance from the head mounted display to the home position;

FIG. 8 is a flowchart illustrating a processing procedure for settingthe home position;

FIGS. 9(a)-9(b) are views for explaining a state where the home positionis set in the head mounted display worn by a user;

FIGS. 10(a)-10(b) are views for explaining a display control of the homeposition;

FIG. 11 is a flowchart illustrating a process for controlling display ofthe home position;

FIGS. 12(a)-12(b) are views for explaining a display example of the homeposition when the home position is set;

FIGS. 13(a)-13(b) are views illustrating an example in a case where thehead mounted display faces a direction different from the home position;

FIG. 14 is a view illustrating an example in which the head mounteddisplay moves an application to the home position;

FIGS. 15(a)-15(b) are views for explaining a state after “call” isselected in a menu;

FIGS. 16(a)-16(b) are views for explaining a case where the user faces adirection at which the home position is located;

FIGS. 17(a)-17(b) are views for explaining a case where the user faces aleft direction again;

FIG. 18 is a flowchart illustrating a processing procedure for a displaycontrol of the application;

FIGS. 19(a)-19(b) are views for explaining an example in which the homeposition is set;

FIGS. 20(a)-20(b) are views for explaining an example a monitor is resetas a home position in a case where the monitor is arranged at a positionof the home position;

FIG. 21 is a flowchart illustrating a processing procedure for linkingthe home position to the monitor; and

FIG. 22 is a flowchart illustrating a processing procedure for a displaycontrol of an application in a case where the monitor is set to the homeposition.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an example of an embodiment of the present invention willbe described with reference to the drawings.

An outline of a head mounted display according to the present embodimentwill be described with reference to FIG. 1(a) to FIG. 1(c), whichillustrate an overview of the head mounted display 100. Specifically,FIG. 1(a) is a top view of the head mounted display 100 in a state wherea user 1 wears the head mounted display 100. FIG. 1(b) is a front viewof the head mounted display 100. FIG. 1(c) is a side view of the headmounted display 100.

The head mounted display 100 is an apparatus that displays anapplication, which the user 1 can visually recognize, and executes theapplication in a state where the user 1 is wearing the head mounteddisplay 100 on his or her head. The head mounted display 100 includes afirst image pickup unit 133 a and a first image pickup unit 133 b eachof which can take an image of an external world. Further, the headmounted display 100 includes a see-through display 131 that is atransmission type of display. Further, as illustrated in FIG. 1(c), thehead mounted display 100 has an audio output unit 141 (for example, anaudio output unit 141 a). Note that a non-transmission type of displaymay be used in place of the see-through display 131. In this case, thehead mounted display 100 displays a result of the taken image on thedisplay.

The head mounted display 100 stores one application or a plurality ofapplications therein, and displays the application on the see-throughdisplay 131 by means of an operation instruction from the user 1. Notethat the head mounted display 100 stores information indicating aposition at which an application is to be displayed (for example, anabsolute position coordinate), and displays the application on the basisof this information.

Subsequently, a visible area indicating an area that can be visuallyrecognized by the user 1 will be described with reference to FIG. 2 .FIG. 2 is a view for explaining the visible area. Here, the visible areameans an angle range that the user 1 can visually recognize through thesee-through display 131. Which area of the external world this visiblearea corresponds to is determined on the basis of which direction thehead mounted display 100 faces with respect to the external world. Theuser 1 changes a direction of his or her head, whereby the user 1 canvisually recognize an arbitrary direction in the visible area. Moreover,the head mounted display 100 arranges virtual objects (for example, adisplay screen, an operation screen, and the like of the applicationdisplayed on the see-through display 131 described above) in a virtualspace that is superimposed on a space around the user 1 in the externalworld and spreads. This makes it possible for the user 1 to visuallyrecognize the virtual object. Further, the user 1 carries out apredetermined operational input against the head mounted display 100,whereby the user 1 can operate the virtual object. The virtual objectsare visually recognized so as to be respectively arranged at specificpositions of the external world. Note that in a case where anon-transmission type of display is used, an area of the image taken bythe first image pickup unit 133 becomes a visible area.

As described above, the head mounted display 100 executes theapplication stored by the head mounted display 100. The head mounteddisplay 100 according to the present embodiment sets an operation area(hereinafter, referred to also as a “home position”) that is an areawhere the user 1 operates the application (for example, an operation foran editing process). Here, positions of the applications and the homeposition are fixed with respect to the external world.

Subsequently, a hardware configuration of the head mounted display 100will be described with reference to FIG. 3 . FIG. 3 is a view forexplaining a hardware configuration of the head mounted display 100. Thehead mounted display 100 includes a main controller 101 to controlvarious kinds of devices via a bus 102. This main controller 101 is aCPU (Central Processing Unit) or the like. Further, the head mounteddisplay 100 includes a RAM 103, a storage 110, a sensor unit 150, acommunication unit 160, and an extension I/F unit 170. The sensor unit150 includes a GPS receiver 151, a gyro sensor 152, a geomagnetic sensor153, and an acceleration sensor 154.

Further, the head mounted display 100 includes an operation input unit120, the see-through display 131, an image signal processor 132, thefirst image pickup unit 133, the second image pickup unit 134, the audiooutput unit 141, an audio signal processor 142, and an audio input unit143. The operation input unit 120 has operation keys 121 and a touchsensor 122. The components described above operate, whereby functions ofthe head mounted display 100 (will be described later) are exerted.

The main controller 101 is an arithmetic processing apparatus thatexecute various kinds of information processing programs stored in thestorage 110 to execute various kinds of functions. The RAM 103 becomes awork area at the time of execution of the various kinds of programs. Thestorage 110 is a part configured to store images that the maincontroller 101 causes the first image pickup unit 133 or the secondimage pickup unit 134 to take and various kinds of information.

The sensor unit 150 is a part configured to obtain information forspecifying a position and a direction of the head mounted display 100.The sensor unit 150 includes the GPS receiver 151, the gyro sensor 152,the geomagnetic sensor 153, and the acceleration sensor 154.

The GPS receiver 151 receives a GPS signal to obtain GPS information.The gyro sensor 152 has a function to detect a moving direction of thehead mounted display 100, and obtains gyro information indicatingangular velocity data accompanying a change in the direction of the headmounted display 100. The geomagnetic sensor 153 is one type of anorientation detector configured to detect an angle indicating absoluteorientation of the head mounted display 100 on the basis of earthmagnetism. As such an angle, for example, an azimuth angle can be cited.The acceleration sensor 154 has a function to detect at least one of anacceleration or an inclined angle of the head mounted display 100.

In this way, the sensor unit 150 can detect at least one of the positionor the direction of the head mounted display 100.

The communication unit 160 is a part configured to execute wirelesscommunication. For example, the communication unit 160 executes nearfield communication. The extension I/F unit 170 is an interface forconnecting to peripheral devices.

The operation input unit 120 is a part configured to receive anoperational input. The operation keys 121 are physical keys installed ata predetermined position of the head mounted display 100 (for example, aportion to be put on his or her ear of the user 1). The touch sensor 122is an input sensor installed on the see-through display 131 or the like.

The see-through display 131 is a see-through type (or a transmissiontype) of display. The image signal processor 132 is a part configured toanalyze the image by the first image pickup unit 133 or the second imagepickup unit 134 to output an analysis result to the main controller 101or the like.

The first image pickup unit 133 is a part configured to take an image ofthe external world. The second image pickup unit 134 is a partconfigured to take an image of an eye of the user 1. The audio outputunit 141 is a speaker that outputs audio or the like. The audio signalprocessor 142 is a part configured to analyze the audio inputted fromthe audio input unit 143. The audio input unit 143 is a part configuredto input audio (or voice) of the user 1, and is a microphone or thelike.

Subsequently, the functions of the head mounted display 100 will bedescribed with reference to FIG. 4 . FIG. 4 is a functional blockdiagram of the head mounted display 100. The head mounted display 100includes a memory 71, a setting receiver 72, a position specifying unit73, an image pickup unit 74, a detector 75, a setting unit 76, and anapplication controller 77 (appli controller).

The memory 71 is a part configured to store various kinds ofinformation. For example, the memory 71 stores a plurality ofapplications therein. Further, the memory 71 may store therein imagedata taken by the image pickup unit 74.

The setting receiver 72 is a part configured to receive an operationarea setting request in response to an operation of the user 1indicating a setting request. When the setting receiver 72 receives anoperation area setting request, the setting receiver 72 notifies theposition specifying unit 73, the image pickup unit 74, and the detector75 of a fact that the operation area setting request is received. Whenthe setting receiver 72 receives the operation area setting request, thehead mounted display 100 shifts to a home position setting mode. Thehome position setting mode means a mode for setting the home position.

Note that instead of receiving an operation by the user 1 indicating thesetting request, the setting receiver 72 may receive an operation areasetting request from a controller (for example, the main controller 101)of the head mounted display 100 in a case where any operation area isnot set at the time of turning on a power source of the head mounteddisplay 100.

Further, the setting receiver 72 is a part configured to receive anoperation area using request in response to an operation by the user 1indicating a usage instruction. When the setting receiver 72 receivesthe operation area using request, the setting receiver 72 notifies theposition specifying unit 73, the image pickup unit 74, and the detector75 of a fact that the azimuth operation area using request is received.When the setting receiver 72 receives the operation area using request,the head mounted display 100 shifts to a home position usage mode. Thehome position usage mode means a mode for using (that is, displaying)the home position.

The position specifying unit 73 is a part configured to specify aposition and a direction (that is, orientation) of the head mounteddisplay 100. The position specifying unit 73 is realized by the sensorunit 150, for example. When the position specifying unit 73 receives,from the setting receiver 72, the fact that the operation area settingrequest described above is received or the fact that the operation areausing request is received, the position specifying unit 73 specifies theposition and the direction of the head mounted display 100. The positionspecifying unit 73 specifies the position and the direction of the headmounted display 100, and sends out the position and the direction thusspecified to the setting unit 76. Further, in case of the home positionusage mode, the position specifying unit 73 sends out the position andthe direction thus specified to the application controller 77.

The image pickup unit 74 is a part configured to take an image of a siteof the user 1 who wears the head mounted display 100. The image pickupunit 74 is realized by the first image pickup unit 133, the second imagepickup unit 134, or the like. When the image pickup unit 74 receivesnotification that the operation area setting request is received fromthe setting receiver 72 or notification that the operation area usingrequest is received, the image pickup unit 74 sends out a result of thetaken image to the detector 75. For example, the image pickup unit 74takes an image of a portion including the visible area described above,or takes an image around the eye of the user 1. The image pickup unit 74sends out the result of the taken image to the detector 75.

The detector 75 is a part configured to detect a position indicated bythe user 1 on the basis of the image taken by the image pickup unit 74.The detector 75 receives notification of reception of the operation areasetting request from the setting receiver 72 or a fact that theoperation area using request is received. When the result of the takenimage is received from the image pickup unit 74, the detector detectsthe position indicated by the user 1.

In a case where the detector 75 obtains an image of a portion includingthe visible area from the image pickup unit 74 as the result of thetaken image, the detector 75 subjects the image to a known imageanalyzing process to specify a motion of a finger included in thevisible area. Namely, the detector 75 detects gesture of the user 1, andspecifies a position based on the gesture.

For example, the detector 75 detects the gesture by the user 1, whichindicates an area, from a result of the image taken by the image pickupunit 74, and specifies an area indicated by the detected result. Thedetector 75 detects the motion of the finger of the user 1, andspecifies the position of the area (an operation area) based on themotion of the finger. The detector 75 sends out information indicatingthe position of the area to the setting unit 76. The informationindicating the position of the area is information based on positionspecified by the finger, for example.

Further, in a case where the detector 75 obtains the taken image aroundthe eye of the user 1 as the image result taken from the image pickupunit 74, the detector 75 subjects the image to the known image analyzingprocess to detect a position in front of a line of sight of the user 1.For example, the detector 75 detects a position in front of the line ofsight on the see-through display 131 as the position in front of theline of sight of the user 1. In this case, the detector 75 sends out theinformation indicating the position to the setting unit 76 asinformation regarding the operation area.

The setting unit 76 is a part configured to set the information (forexample, a position regarding the operation area) regarding theoperation area (that is, the home position), which is an area foroperating an application stored in the memory 71, on the basis of theresult detected by the detector 75 and the position specified by theposition specifying unit 73.

In a case where the setting unit 76 obtains the position and thedirection specified by the position specifying unit 73 and theinformation indicating the area indicated by the result detected by thedetector 75, the setting unit 76 sets, to the memory 71, the informationindicating the area and the position and the direction specified by theposition specifying unit 73.

The setting unit 76 sets, to the memory 71, the position and thedirection specified by the position specifying unit 73, and theinformation on the area defined on the basis of the position in front ofthe line of sight by the result detected by the detector 75 or theposition of the area based on the motion of the finger.

Further, in a case where the setting unit 76 specifies a monitorcorresponding to the position based on the result detected by thedetector 75, the setting unit 76 may set a position of the monitor tothe position of the operation area. For example, the setting unit 76obtains, from the image pickup unit 74, an image corresponding to theposition detected by the detector 75 and compares the image with atemplate of the monitor stored in advance to detect the monitor, thesetting unit 76 sets identification information of the monitor as theinformation regarding the operation area.

The application controller 77 is a part configured to display, inresponse to an operation request by the user 1, an application stored inthe memory 71 in the operation area of the see-through display 131. In acase where an operation request for editing a predetermined applicationis made by the operation request by the user 1, the applicationcontroller 77 displays the application in the operation area, and setsthe application to an editable state.

Further, in a case where information regarding the home position hasalready been stored in the memory 71, the application controller 77appropriately obtains the position and the direction of the head mounteddisplay 100 from the position specifying unit 73, and determines whetherthe home position is included in the visible area or not on the basis ofthe position and the direction. In a case where it is determined thatthe home position is included in the visible area, the applicationcontroller 77 outputs the area of the home position to the see-throughdisplay 131.

Further, in a case where the information regarding the operation areahas already been set and the monitor corresponding to the position isspecified, the application controller 77 resets the monitor to aposition of a new operation area.

Further, the application controller 77 stores a display position of eachof the applications, and displays the application on the basis of thecorresponding display position. The display position is a positiondefined by a position, a direction, and the like of the user 1. Notethat the memory 71 may store the display positions of the applicationstherein.

<Setting of Home Position>

Subsequently, a method of setting a home position will be described withreference to FIGS. 5(a)-5(c). As a premise, the head mounted display 100becomes a setting mode for a home position on the basis of an operationreceived by the operation input unit 120.

When the head mounted display 100 becomes the setting mode for the homeposition as described above, the image pickup unit 74 takes an image ofa visible area illustrated in FIG. 5(a) for a predetermined period, andtakes an image of a finger of the user 1 as illustrated in FIG. 5(a).For example, the detector 75 detects a track of the finger from an uppercentral portion of a rectangular area to a lower central portion of therectangular area.

Namely, the detector 75 detects a motion (gesture) of the finger asillustrated in FIG. 5(a) by using an image result taken by the imagepickup unit 74; specifies a designation area (the rectangular area) bythe user 1 on the basis of the motion of the finger; and sets thedesignation area as the home position. Moreover, the detector 75determines a reference position of the home position on the basis of adifference between the visible area and an area of the see-throughdisplay 131.

For example, the detector 75 determines the reference position of thehome position in the see-through display 131 on the basis of apositional relationship in the visible area and a difference in a sizebetween the visible area and the area of the see-through display 131.Here, a method of determining the reference position described abovewill be described with reference to FIG. 5(b). FIG. 5(b) illustrates adisplay area in the see-through display 131.

Further, the detector 75 calculates a ratio among a height V, a distanceVU, a distance VP, and a distance VL. The height V is a height of thesee-through display 131. The distance VU is a distance from an upperportion of the see-through display 131 to an upper portion of the homeposition. The distance VP is a distance from the upper portion of thehome position to a lower portion of the home position. The distance VLis a distance from the lower portion of the home position to a lowerportion of the see-through display 131.

Further, the detector 75 calculates a ratio among a width H of thesee-through display 131, a distance HL from a left portion of thesee-through display 131 to a left portion of the home position, adistance HP from the left portion of the home position to a rightportion of the home position, and a distance HR from the right portionof the home position to a right portion of the see-through display 131.

The detector 75 calculates a distance from the head mounted display 100to a position of the finger, and defines a reference position P on thebasis of the distance and the calculated ratio described above. Thisreference position P indicates a position to be displayed in thesee-through display 131. By combining a current position and orientationspecified by the position specifying unit 73, it is possible to definean absolute position of the home position in the external world. Theabsolute position in the external world obtained herein is basically thehome position, but as a modification example, a specific position in thevisible area illustrated in FIG. 5(b) may be set to the home position.In case of this modification example, it is possible for the user 1 toalways recognize the home position visually.

Note that the detector 75 may specify the position in the specified areawithout considering the ratio as described above.

FIG. 5(a) illustrates the example in a case where a line from the uppercentral portion of the home position to the lower central portion of thehome position is traced with the finger. However, the home position maybe specified by another method. Here, another method of specifying ahome position will be described with respect to FIGS. 6(a)-6(c). Forexample, as illustrated in FIG. 6(a), the home position may be specifiedalong an area of the home position from an upper right portion of thehome position to a lower left portion thereof. Further, as illustratedin FIG. 6(b), the home position may be specified along a diagonal lineof the area of the home position by a line from the upper right portionof the home position to the lower left portion thereof.

Further, the detector 75 calculates a distance from the head mounteddisplay 100 to a reference position of the home position on the basis ofa result of the taken image. A method of calculating the distance fromthe head mounted display 100 to the home position will be described withreference to FIG. 7(a) and FIG. 7(b). FIG. 7(a) is a top view of thehead mounted display 100 in a state where the user 1 wears the headmounted display 100, and is a view illustrating a horizontal angle. FIG.7(b) is a side view of the head mounted display 100, and is a viewillustrating an elevation/depression angle.

Specifically, the detector 75 calculates a horizontal angle α of areference position of a home position AR with respect to orientation ofthe head mounted display 100 illustrated in FIG. 7(a) on the basis of aposition of a finger of the taken image. Further, the detector 75calculates an elevation/depression angle R of the reference position ofthe home position AR with respect to the orientation of the head mounteddisplay 100 illustrated in FIG. 7(b) on the basis of the position of thefinger of the taken image.

The detector 75 calculates a distance L to the reference position of thehome position AR on the basis of a distance to a central portion of ataken image range, the horizontal angle α and the elevation/depressionangle β described above.

Note that the detector 75 sends out, to the setting unit 76, thedistance L from the head mounted display 100 to the reference positionof the home position AR, a width and a height of the home position AR,and the reference position of the home position AR. In response to this,the setting unit 76 stores the distance L from the head mounted display100 to the reference position of the home position AR, the width of thehome position, the height of the home position, and the referenceposition P of the home position in the memory 71 as informationregarding the home position. Note that needless to say, an absolutecoordinate of the reference position P of the home position can becalculated from the position and the orientation of the head mounteddisplay 100 obtained by the position specifying unit 73, and thehorizontal angle α, the elevation/depression angle β, and the distance Ldescribed above.

Here, a procedure for setting the home position AR will be describedwith reference to FIG. 8 . FIG. 8 is a flowchart illustrating aprocessing procedure for setting the home position AR. First, when thesetting receiver 72 receives an operation area setting request inaccordance with an operation by the user 1 indicating a setting request,the head mounted display 100 shifts to a home position setting mode(Step S1: Yes). Note that the processing is terminated in a case wherethe setting receiver 72 does not receive any operation area settingrequest (Step S1: No).

When the setting receiver 72 receives the operation area settingrequest, the setting receiver 72 notifies the position specifying unit73, the image pickup unit 74, and the detector 75 of a fact that theoperation area setting request is received. In response to this, theimage pickup unit 74 starts (or enables) an image taking operation, andsends out a result of the taken image to the detector 75 (Step S2).

The position specifying unit 73 obtains a current position of the headmounted display 100 by the GPS receiver 151 or the like (Step S3).Further, the position specifying unit 73 obtains orientation (that is, adirection in which the head mounted display 100 is facing) by thegeomagnetic sensor 153 or the like (Step S4). Further, the detector 75obtains the result of the taken image from the image pickup unit 74;specifies a motion of the user 1 on the basis of the result of the takenimage; and calculates a distance to a finger of the user 1 on the basisof the result of the taken image (Step S5). Further, the detector 75specifies a height and a width of home position on the basis of theresult of the taken image described above (Step S6). Further, thedetector 75 calculates a reference position P on the basis of thedistance to the finger of the user 1. The detector 75 sends out, to thesetting unit 76, the distance L from the head mounted display 100 to thereference position of the home position AR, the width of the homeposition, the height of the home position, and the reference position Pof the home position (Step S7).

When the setting unit 76 obtains the reference position P and the heightand the width of the home position AR described above, the setting unit76 displays a confirmation message of the home position AR on thesee-through display 131, and confirms the setting (Step S8).

When an input indicating approval of the setting is made by anoperational input by the user 1 (Step S8: Yes), the setting unit 76 setsthe distance L, the height and the width of the home position AR, andthe reference position P to the memory 71. Further, the setting unit 76also sets the current position and the orientation specified by theposition specifying unit 73 to the memory 71 as information regardingthe home position AR (Step S9).

Further, the setting unit 76 notifies the image pickup unit 74 of thatthe setting is completed, and the image pickup unit 74 turns off (ordisables) an image taking state (Step S10). Further, the setting unit 76releases the home position setting mode. Note that when an inputindicating denial of the setting is made at Step S8 by the operationalinput by the user 1 (Step S8: No), the processing flow shifts to StepS3.

Subsequently, a display example of the home position AR at the time ofmovement of the user 1 will be described with reference to FIGS.9(a)-9(b) and FIGS. 10(a)-10(b). First, a state where the home positionAR is set in the head mounted display 100 worn by the user 1 will bedescribed with reference to FIGS. 9(a)-9(b).

FIG. 9(a) is a floor plan of a room of the user 1. As illustrated inFIG. 9(a), a bed 600, a desk 601, a chair 602, a book shelf 603, a TVstand 604, and a TV 605 are included in the room of the user 1.

FIG. 9(b) is a view illustrating a state where the user 1 is sitting onthe chair 602. The user 1 wearing the head mounted display 100 operatesthe operation input unit 120 in a state of sitting on the chair 602, andthe head mounted display 100 sets information regarding the homeposition AR in accordance with this.

Subsequently, a display control of the home position AR will bedescribed with reference to FIGS. 10(a)-10(b). FIG. 10(a) is a viewillustrating an example in which the user 1 is looking at a placedifferent from the home position (that is, the head mounted display 100is facing a direction different from the home position). As illustratedin FIG. 10(a), there is not the reference position P in the direction ofthe head mounted display 100 (that is, in a visible area), the headmounted display 100 does not display the home position AR.

Further, FIG. 10(b) illustrates an example in which the head mounteddisplay 100 is facing a direction of the home position AR (that is,there is the reference position P in the visible area). As illustratedin FIG. 10(b), in a case where the head mounted display 100 faces thedirection of the home position AR, the reference position P is includedin the visible area. Therefore, the head mounted display 100 displaysthe home position AR. A display mode of the home position at this timemay be configured so that the direction at the time of the setting ismaintained, or may be configured so that the direction is changed into adirection of the user 1.

Subsequently, a processing procedure for controlling display of the homeposition AR will be described with reference to FIG. 11 . FIG. 11 is aflowchart illustrating a process of controlling display of the homeposition AR. In a case where a home position usage mode is specified byan operational input of the user 1 (Step S11: Yes), the applicationcontroller 77 obtains information regarding the home position AR storedin the memory 71 (containing a current position, orientation, a distanceL, a width and a height of a home position, and a reference position P)(Step S12).

Subsequently, the application controller 77 obtains information on acurrent position from the position specifying unit 73 (Step S13).Further, the application controller 77 obtains orientation from theposition specifying unit 73 (Step S14).

The application controller 77 compares the information regarding thehome position AR stored in the memory 71 with the information on thecurrent position and the orientation specified by the positionspecifying unit 73. In a case where the reference position P of the homeposition is within a visible area as a result of the comparison (StepS15: Yes) and a distance from the head mounted display 100 to thereference position P of the home position is within a predeterminedvalue (Step S16: Yes), the application controller 77 displays the homeposition on the see-through display 131 (Step S17). Note that in a casewhere the condition at Step S15 is not satisfied (Step S15: No) or thecondition at Step S16 is not satisfied (Step S16: No), the processingflow proceeds to Step S18.

<Movement of Application to Home Position>

Subsequently, a method of moving an application displayed by the headmounted display 100 to a home position will be described with referenceto FIGS. 12(a)-12(b) to FIGS. 17(a)-17(b).

A display example of the home position at the time of setting the homeposition will first be described with reference to FIGS. 12(a)-12(b).FIG. 12(a) is a view illustrating a state where the user 1 is sitting onthe chair 602. In this state, the head mounted display 100 sets the homeposition in response to an operation request from the user 1.

FIG. 12(b) is an example of an image displayed on the see-throughdisplay 131 of the head mounted display 100 in this state. Asillustrated in FIG. 12(b), the head mounted display 100 displays anapplication AP1 (an application A) and an application AP2 (anapplication B), and further displays the home position on thesee-through display 131.

Subsequently, FIGS. 13(a)-13(b) illustrate an example in a case wherethe head mounted display 100 is facing a direction different from thatof the home position. FIG. 13(a) is a view illustrating an example inwhich the head mounted display 100 is facing the direction differentfrom that of the home position. FIG. 13(b) is a view illustrating anexample of an image displayed on the see-through display 131 of the headmounted display 100 in the state illustrated in FIG. 13(a).

As illustrated in FIG. 13(b), the head mounted display 100 does notdisplay the home position, but displays, on the see-through display 131,an application AP3 (an application C), an application AP4 (anapplication D), an application AP5 (an application E), an applicationAP6 (an application F), which are assigned into the direction.

Subsequently, FIG. 14 illustrates an example in which the head mounteddisplay 100 moves the application to the home position. As illustratedin FIG. 14 , when the application AP5 is selected in the stateillustrated in FIG. 13(b) by an operational input by the user 1, thehead mounted display 100 displays a menu. In this menu, “call” isselected. This “call” is a menu indicating invoke or call a target tothe home position.

A state after the “call” is selected in the menu will be described withreference to FIGS. 15(a)-15(b). As illustrated in FIG. 15(a), adirection that the user 1 is facing does not change as well as that inFIG. 13(a). FIG. 15(b) is a view illustrating a display state of thesee-through display 131 after the menu is selected. As illustrated inFIG. 15(b), the application AP5 that has been selected and to which the“call” is specified moves to the home position. Therefore, theapplication AP5 is excluded from display targets. In this case, theapplication controller 77 stores a position before movement and aposition after movement of the application AP5. Note that after theapplication AP5 is called, the head mounted display 100 may display aportion of the application AP5 before movement with a lighter color thanthat before movement. This makes it possible for the user 1 to visuallyrecognize the position of the application AP5 before movement, and theuser 1 can refer to the position when a display position of anapplication is adjusted in this area of the application AP5 beforemovement.

Subsequently, a case where the user 1 faces a direction in which thehome position is positioned will be described with reference to FIGS.16(a)-16(b). FIG. 16(a) is a view illustrating a state where the user 1changes his or her posture and the head mounted display 100 faces adirection in which the home position is included in the visible area.

FIG. 16(b) is a view illustrating an example of an image displayed onthe see-through display 131 of the head mounted display 100 in the statedescribed above. As illustrated in FIG. 16(b), the applicationcontroller 77 arranges and displays the application AP5 at a positionwithin the home position AR. Further, in a case where the applicationcontroller 77 receives a menu display request for the application AP5 byan input operation of the user 1, the application controller 77 displaysthe menu. When “return” is selected, the application controller 77excludes the application AP5 from the display targets.

Subsequently, a case where the user 1 faces a left direction again willbe described with reference to FIGS. 17(a)-17(b). FIG. 17(a) is a viewillustrating a state where the user 1 changes his or her posture and thehead mounted display 100 faces the left direction. FIG. 17(b) is a viewillustrating an example of an image displayed on the see-through display131 of the head mounted display 100 in the state described above.

As illustrated in FIG. 17(b), the application controller 77 displays theapplication AP5 so as to be returned to an original position thereof ina case where “return” for the application AP5 is selected from the menu.

Subsequently, a processing procedure for a display control of anapplication will be described with reference to FIG. 18 . FIG. 18 is aflowchart illustrating a processing procedure for a display control ofan application. First, in a case where the head mounted display 100 isin the home position usage mode (Step S21: Yes), the position specifyingunit 73 obtains a current position of the head mounted display 100 (StepS22). Further, the position specifying unit 73 obtains orientationthereof (Step S23). The application controller 77 obtains the currentposition and the orientation described above from the positionspecifying unit 73, and displays an application based on the currentposition and the orientation on the basis of a stored display positionof the application (Step S24).

In a case where the application controller 77 detects that anyapplication is selected (Step S25: Yes), the application controller 77displays a menu (Step S26). The application controller 77 specifies aselected command (Step S27). In a case where the specified command is“return”, the application controller 77 causes the selected applicationto move to an original position (Step S29), the processing flow proceedsto Step S30. Further, in a case where the specified command is “call”,the application controller 77 moves the selected application to the homeposition (Step S28), the processing flow proceeds to Step S30. Further,in a case where the application controller 77 selects a menu other than“call” and “return”, the processing flow proceeds to Step S30 withoutexecuting anything. At Step S30, in a case where an instructionindicating termination of the home position usage mode is not made (StepS30: No), the processing flow proceeds to Step S22. Further, in a casewhere the instruction indicating termination of the home position ismade at Step S30 (Step S30: Yes), the processing procedure isterminated.

<Process of Linking to Monitor>

Subsequently, an example of linking the monitor to the home position ARin a case where there is the monitor at a position of the home positionAR will be described. An example in which the home position AR is setwill first be described. FIG. 19(a) is a view illustrating a state wherethe user 1 is sitting on the chair 602. The user 1 who wears the headmounted display 100 operates the operation input unit 120 in a state ofsitting on the chair 602, the home position AR is set in response tothis. FIG. 19(b) is a view illustrating an example of an image displayedon the see-through display 131 of the head mounted display 100 in thestate described above. As illustrated in FIG. 19(b), the applicationcontroller 77 displays the home position AR on the see-through display131.

Subsequently, an example of resetting the monitor 700 as the homeposition AR in a case where a monitor 700 is arranged at a position ofthe home position AR will be described with reference to FIGS.20(a)-20(b).

FIG. 20(a) is a view illustrating an example in which the monitor isarranged in front of the user 1 in a state where the user 1 is sittingon the chair 602.

In such a state, the application controller 77 determines whether themonitor is arranged in a range of the home position or not. For example,the application controller 77 compares a result of an image taken by theimage pickup unit 74 with the template of the monitor stored in advanceto determine whether the monitor exists or not.

In a case where it is determined that the monitor exists, theapplication controller 77 displays a message for inquiring whether to beconnected to the monitor or not on the see-through display 131. When anoperational input of the user 1 is made to connect to the monitor, theapplication controller 77 executes near field communication with themonitor. When the monitor is set to a state where information can betransmitted to or received from the monitor, the application controller77 obtains identification information of the monitor, and adds theidentification information of the monitor as information regarding thehome position AR. In this way, the application controller 77 serves as asetting modifier.

Then, as illustrated in FIG. 20(b), the application controller 77displays the monitor as the home position AR.

Further, as a modification example, in a case where the monitor 700 canbe visually recognized in the visible area, a method of setting a screenof the monitor 700 as a new home position may be adopted. In that case,it may be possible to return to an original home position setting thatis not linked to the monitor if necessary.

Subsequently, a processing procedure for linking the home position AR tothe monitor will be described with reference to FIG. 21 . FIG. 21 is aflowchart illustrating a processing procedure for linking the homeposition AR to the monitor 700. Since Steps S31 to Step S35 illustratedin FIG. 21 are respectively the same as Steps S11 to Step S15illustrated in FIG. 11 , their explanation is omitted.

At Step S36, the application controller 77 compares a result of an imagetaken by the image pickup unit 74 with a template of the monitor 700stored in advance to determine whether the monitor 700 exists in a rangeof the home position or not (Step S36). In a case where it is determinedthat the monitor 700 exists in the range of the home position AR (StepS36: Yes), the application controller 77 outputs a message for inquiringwhether to link with the monitor 700 or not (Step S37). In a case wherelink release is specified by a user operation, the applicationcontroller 77 releases link with the monitor 700 (Step S40), theprocessing flow proceeds to Step S41. Note that in a case where it isdetermined that the monitor 700 does not exist in the range of the homeposition AR (Step S36: No), the processing flow proceeds to Step S41.

At Step S37, in a case where a user operation for linking with themonitor is made (Step S37: Yes) and there is a plurality of linkablemonitors 700, the application controller 77 displays a list of themonitors 700, and receives selection of any monitor 700 (Step S38). Notethat in a case where there is only one monitor 700, the process at StepS38 may be omitted. The application controller 77 links with theselected monitor 700 (resets the home position to the monitor) (StepS39), the processing flow proceeds to Step S41.

At Step S41, in a case where an instruction for terminating the homeposition usage mode is made (Step S41: Yes), the processing procedure isterminated. Further, in a case where no instruction for terminating thehome position usage mode is made at Step S41 (Step S41: No), theprocessing flow proceeds to Step S33.

Subsequently, a processing procedure of a display control of theapplication in a case where the monitor 700 is set to the home positionwill be described with reference to FIG. 22 . FIG. 22 is a flowchartillustrating a processing procedure of a display control of theapplication in a case where the monitor 700 is set to the home positionAR. Note that since Step S51 to Step S57, and Step S62 illustrated inFIG. 22 are respectively the same as Step S21 to Step S27, and Step S30illustrated in FIG. 18 , their explanation is omitted.

At Step S57, in a case where the specified command is “return”, there isno target to be displayed on the monitor 700. Therefore, the applicationcontroller 77 terminates an output of a selected application to amonitor (Step S60); move the application to an original position (StepS61); and the processing flow proceeds to Step S62.

Further, at Step S57, in a case where the specified command is “call”,the application controller 77 terminates display of the selectedapplication (Step S58) and displays the application on the monitor 700that is the home position (Step S59), and the processing flow proceedsto Step S62. Further, in a case where any menu other than “call” and“return” is selected, the application controller 77 causes theprocessing flow to proceed to Step S62 without doing anything.

As described above, in the head mounted display 100, the memory 71stores applications therein. The image pickup unit 74 takes an image ofa site of the user 1, and the position specifying unit 73 specifies theposition and the direction of the head mounted display 100. The detector75 detects a state of the user 1 on the basis of the image taken by theimage pickup unit 74, and the setting unit 76 sets a position indicatingthe home position on the basis of the result detected by the detector 75and the position and the direction specified by the position specifyingunit 73.

The head mounted display 100 sets the area where the application isoperated by means of input specification of the user 1. Therefore, theuser 1 can operate the application at a position where the user 1 canoperate the application easily.

Specifically, the image pickup unit 74 takes an image of a portionincluding the finger of the user 1; the detector 75 detects gesture ofthe user 1 on the basis of the result of the taken image; and thesetting unit 76 sets a position regarding the home position by using theposition by the gesture. In this case, the head mounted display 100 setsthe position regarding the home position on the basis of a result of afinger motion (or gesture) of the user 1. Therefore, it is possible toset the home position by an intuitive and simple operation without usinga complicated operational input.

Further, the image pickup unit 74 takes the image around the eye of theuser 1; the detector 75 detects the line of sight of the user by usingthe image result taken by the image pickup unit 74, and detects theposition by the line of sight; and the setting unit 76 sets the positionregarding the home position by using the position by the line of sight.In this case, the head mounted display 100 can also set the homeposition by the intuitive and simple operation without using acomplicated operational input.

Further, in a case where there is the monitor 700 at the position basedon the position detected by the detector 75 and the position specifiedby the position specifying unit 73, the setting unit 76 sets theposition of the monitor 700 at the position regarding the home positionAR. In this case, the head mounted display 100 can sets the monitor 700as the home position AR by an intuitive operation such as the gesture ofthe user 1.

Further, in a case where the position regarding the home position AR isset by the setting unit 76 and the application controller 77 newlyspecifies the monitor 700 at a position corresponding to the position,the application controller 77 newly resets the position of the monitoras the home position. As a result, after setting the home position, thehead mounted display 100 can set the position of the monitor to the homeposition without requiring an operation indicating specification of anew position by the user 1.

Further, the application controller 77 controls, in response to apredetermined input operation, so that the application stored in thememory 71 is displayed at the home position, whereby it is possible todisplay the application at a position where the user operates theapplication easily.

Although it has not been mentioned particularly in the embodimentdescribed above, a plurality of home positions may be set. Further, inthis case, a home position as a target may be determined on the basis ofa current position of the head mounted display 100, or a home positionspecified by a user operation may be prioritized.

The case where the position specifying unit 73 specifies the position onthe basis of the current position such as latitude or longitude has beendescribed. However, the position may be specified by any of variouskinds of other methods. For example, a relative position based on areference object (an object that becomes a landmark) may be specified.

Although it has not mentioned particularly in the embodiment describedabove, command specification (for example, “call” or the like) may bespecified by an audio input.

In the embodiment described above, the case where the distance L, thewidth and the height of the home position, and the reference position Pare set as the information regarding the home position in addition tothe current position and the orientation has been described. However,with respect to the distance L, and the width and the height of the homeposition, values defined in advance may be held, and the currentposition, the orientation, and the reference position P may be set.

A part or all of the functions and the like described above of thepresent invention may be realized by hardware that is designed by anintegrated circuit, for example. Further, the functions and the like mayalso be realized by software so that a microprocessor unit or the likeinterprets operation programs realizing the respective functions andexecutes the interpreted operation programs. Hardware and software maybe used together.

Further, control lines and information lines illustrated in figures areillustrated so long as they are thought to be necessary for explanation.All of the control lines and the information line are not necessarilyillustrated on a product. In fact, it may be considered that almost allof the components are connected to each other.

Further, control lines and information lines illustrated in figures areillustrated so long as they are thought to be necessary for explanation.All of the control lines and the information line are not necessarilyillustrated on a product. In fact, it may be considered that almost allof the components are connected to each other.

REFERENCE SINGS LIST

1 . . . user, 71 . . . memory, 72 . . . setting receiver, 73 . . .position specifying unit, 74 . . . image pickup unit, 75 . . . detector,76 . . . setting unit, 77 . . . application controller, 100 . . . headmounted display, 101 . . . main controller, 103 . . . RAM, 110 . . .storage, 120 . . . operation input unit, 121 . . . operation keys, 122 .. . touch sensor, 131 . . . see-through display, 132 . . . image signalprocessor, 133 . . . first image pickup unit, 134 . . . second imagepickup unit, 141 . . . audio output unit, 142 . . . audio signalprocessor, 143 . . . audio input unit, 150 . . . sensor unit, 151 . . .GPS receiver, 152 . . . gyro sensor, 153 . . . geomagnetic sensor, 154 .. . acceleration sensor, 160 . . . communication unit, and 170 . . .extension I/F unit.

1. A virtual object operating method executed by a head mounted displaycapable of visually recognizing a virtual object arranged at anarbitrary position in a virtual space, the virtual object operatingmethod comprising: an operation receiving step of receiving anoperational input for operating the virtual space or the virtual object;an external-display cooperating step of causing an external displayapparatus in a practical space to cooperate as an apparatus foroperating the virtual object in a case where an external-displaycooperating operation is received in the operation receiving step; avirtual object selecting step of selecting a predetermined virtualobject from virtual objects arranged at an arbitrary position in thevirtual space in a case where a virtual object selecting operation isreceived in the operation receiving step; and a virtual objectoutputting step of outputting information regarding the predeterminedvirtual object selected in the virtual object selecting step to theexternal display being caused to cooperate in the external-displaycooperating step in a case where a virtual object editing requestoperation is received in the operation receiving step.
 2. The virtualobject operating method according to claim 1, further comprising: anexternal-display confirming step of confirming whether the externaldisplay apparatus exists at a position of the practical spacecorresponding to the specified position of the virtual space specifiedby an operation-area setting operation in a case where theoperation-area setting operation is received in the operation receivingstep, wherein, in the external-display cooperating step, the externaldisplay apparatus is caused to cooperate as an apparatus for operatingthe virtual object in a case where it is confirmed in the displayconfirming step that the external display apparatus exists.
 3. Thevirtual object operating method according to claim 1, furthercomprising: a virtual object ouptput ending step of ending output of theinformation regarding the selected predetermined virtual object to theexternal display being caused to cooperate in the external-displaycooperating step, the virtual object being selected in the virtualobject selecting step, in a case where a virtual object editing endoperation is received in the operation receiving step.
 4. A head mounteddisplay capable of visually recognizing a virtual object arranged at anarbitrary position in a virtual space, the head mounted displaycomprising: an operation receiver configured to receive an operationalinput for operating the virtual space or the virtual object; acommunication unit performing transmission/receiption of data to/fromoutside; and a controller, wherein the controller is configured to:cause an external display apparatus in a practical space to cooperate asan apparatus for operating the virtual object in a case where theoperation receiver receives an external-display copperation operation;select a predetermined virtual object from virtual objects arranged atan arbitrary position in the virtual space in a case where the operationreceiver receives a virtual object selecting operation; and outputinformation regarding the selected predetermined virtual object throughthe communication unit to the external display apparatus being caused tocooperate in a case where the operation receiver receives a virtualobject editing request operation.
 5. The head mounted display accordingto claim 4, wherein the controller is further configured to confirmwhether the external display apparatus exists at a position of thepractical space corresponding to the specified position of the virtualspace specified by an operation-area setting operation in a case wherethe operation receiver receives the operation-area setting operation,and causing the external display apparatus to cooperate as an apparatusfor operating the virtual object in a case where it is confirmed thatthe external display apparatus exists.
 6. The head mounted displayaccording to claim 4, wherein the controller is further configured toend output of the information regading the selected predeterminedvirtual object through the communication unit to the external displaybeing caused to cooperate in a case where the operation receiverreceives a virtual object editing end operation.